Today, the typical setting for a personal computer system is still an office. Since personal computers started becoming prevalent over twenty years ago, many of them have been used for such applications as word processing (and other document preparation), financial calculations, and other office-related uses. They have not permeated the home environment, other than for games and for displaced office-type work, because they are not simple to operate.
During that time, the primary user-interface paradigm for interacting with computers has been a keyboard-and-screen system. Although this arrangement has been improved and refined over the years, it is still essentially the same arrangement that has been used with computers for many years, and was used on remote terminals even before the advent of personal computers.
The keyboard-and-screen system presents several advantages. The keyboard typically used with computer systems is an inexpensive item to produce. It includes only around 100 key switches, signals from which are encoded and sent to the CPU. Also, it is only slightly modified from the version used on mechanical typewriter keyboards for over a century. Hence, it is familiar to most people. Moreover, typewriter-style keyboards (or variations thereof) are usable to unambiguously input information in most Western languages.
However, for most people, a keyboard is not an efficient form of input. To use a keyboard effectively, training is required. Even with the requisite training, time and effort is necessary to enter information via keyboard, particularly when the information sought to be entered is already evident in a document or some other communication from another. Moreover, they are sensitive to spelling errors, repetitive stress injury (such as carpal tunnel syndrome and tendinitis), and inconvenience. Both hands are needed to use a traditional keyboard with any speed.
The display screens (or "monitors") typically used on personal (and other) computer systems have a number of advantages. They are relatively inexpensive. Years of producing television sets and computer monitors have resulted in manufacturing and design efficiencies and improvements in quality.
However, even with their improvements, CRT-based display screens are still typically bulky, heavy, and energy inefficient. They also produce a relatively large amount of heat. For these reasons, CRT displays have not been integrated into many other environments, and computers (and their displays) are usually treated as stand-alone items. Other display technologies have been tried, including plasma displays and liquid crystal displays (LCDs), to name two, but have been less successful because of their relatively high cost and low image quality in comparison to CRT displays. However, LCD prices have been dropping in recent years, and such displays are beginning to be found in a variety of applications.
While the keyboard-and-screen scheme for interacting with computers has proven to be satisfactory in many ways for a long time, there are some problems that are not easily resolved with such a system. For example, there can be a lack of correlation between what is displayed on the screen and what is entered on the keyboard. Any formatting information available on the screen must be entered via sequences of keystrokes on the keyboard, and those sequences in many cases are not intuitive. Furthermore, many symbols and items viewable on the screen can not easily be entered via keyboard.
Recently, however, progress has been made in the usability of alternative user interface schemes. For example, touch-screen-based systems, in which a flat-panel display (such as an LCD) is overlaid with a translucent pressure-sensitive (or other type of touch-sensitive) surface, have been gaining in popularity. Such a system allows the user to directly manipulate the information that is shown on the display. For example, various gestures can be made on the surface to copy, move, annotate, or otherwise alter information on the display. Where such a system falls short, however, is in data input. Where there is no keyboard associated with a touch screen, then data must be input via interaction with the touch-sensitive surface. In some cases, this involves handwriting recognition, which is an imperfect and computationally intensive procedure, or some other means, such as "pressing" (with a stylus or fingertip) a visually displayed keyboard, or by special "gestural symbols" designed for data entry.
Speech recognition input has also made some progress in recent years. In the past, speech recognition systems have been used primarily in experimental environments. Typically, error rates were extremely high, and to accomplish real-time recognition, the computational resources required were prohibitively high. Recently, however, several commercial software products have made it possible to offer real-time speech recognition on personal computers of the type frequently used in the home. However, such speech recognition systems are speaker-dependent, and as such require a significant amount of training to attain a satisfactory level of performance and a low enough error rate. Moreover, when errors are made (such as in the recognition of homonyms and proper names), it is frequently more convenient to type the corrected word with a traditional keyboard than it is to correct the error by speaking the necessary voice commands and spelling the word for the system. Accordingly, speech recognition shows some promise for the future, but at the present time, is not a practical method of operating and providing input to a personal computer.
Despite promises of cross-platform integration (e.g., computer and television, computer telephony), there is usually little relationship between the data on a personal computer and most of the document and other tools used for communication and information exchange that are found around a typical individual, office, or family. For example, in a typical home or office, one might find a telephone, an answering machine (or voicemail system), audio equipment (such as a stereo), a fax machine, a television, a computer and printer, a whiteboard or a chalkboard, and various written notes, lists, calendars, mailings, books, and other documents. Unfortunately, the information in one or more of those repositories is usually tied to that repository. For example, addresses in a written address book are not easily used on a computer e-mail system, unless the user goes to the trouble of manually transferring the relevant information from the address book to the computer.
Furthermore, there is a well-known lack of compatibility between systems of different types, even those systems that are designed to work together. For example, in the conversion between one data format and another, there may be a loss of formatting or other information. Furthermore, errors may creep into the conversion, as when optical character recognition (OCR) is used to convert a printed document to a machine-readable one.
Because of these obstacles, the numerous disparate data types and formats persist in the home and office environments. For example, written notes on a family's refrigerator door are frequently a useful and convenient means of communication. The kitchen is often a place of gathering, or at least a place where each family member will visit several times every day. Accordingly, when one family member wishes to communicate with another that he might not see in person, then he might write a short note and post it to the refrigerator door with, for example, a magnet. Other documents, such as calendars, computer printouts, facsimiles, and collaborative lists can also be posted to the refrigerator door.
Several companies have introduced limited-function kitchen computer, or software for general-purpose personal computers to enable kitchen functionality. Such kitchen computers usually provide the ability to store recipes, create shopping lists, and take rudimentary notes. However, in most cases, these kitchen computers use the standard keyboard-and screen user interface, and are highly limited in function. Kitchen computers generally do not have very well-developed document handling or telephony functions.
A class of portable computers known as personal document readers (or PDRs) has also arisen in recent years. The goal of these devices is to serve as a replacement for a printed book. Accordingly, a typical PDR is relatively light in weight, but has a large high-contrast screen to enable easy reading. Recent PDRs also have other capabilities, such as the ability to annotate a document (either via a built-in keyboard or a touch-sensitive screen adapted for writing with a stylus). Known PDRs are generally limited in function, but as will be discussed below, can frequently be used as an input/output terminal in an embodiment of the invention to be disclosed herein.
Another class of systems uses what is known as a "paper user interface," in which commands are conveyed to the system by making marks on paper, which is then scanned. For example, one category of such devices is able to read free-form ink (or digital) annotations to determine which of several possible editing operations a user wishes to perform on a document. See, e.g., U.S. Pat. No. 5,659,639 to Mahoney and Rao, entitled "Analyzing an Image Showing Editing Marks to Obtain Category of Editing Operation." Other versions of "paper UI" systems are capable of interpreting drawn symbols as commands, deriving commands from marked-up forms (e.g., checkboxes) attached to a scanned document, and reading pre-printed one- or two-dimensional data codes (such as Xerox DATAGLYPHS). For a summary of the state of the art in this area, see, for example, U.S. Pat. No. 5,692,073 to Cass, entitled "Formless Forms and Paper Web Using a Reference-Based Mark Extraction Technique."
Such techniques can also be extrapolated to other media, such as office whiteboards. See, e.g., U.S. Pat. No. 5,528,290 to Saund, entitled "Device for Transcribing Images on a Board Using a Camera Based Board Scanner," and U.S. Pat. No. 5,581,637 to Cass and Saund, entitled "System for Registering Component Image Tiles in a Camera-Based Scanner Device Transcribing Scene Images." An all-electronic system for accomplishing essentially the same result is Tivoli, an electronic collaboration tool that uses Xerox LIVEBOARD hardware. See Pedersen, E. R., McCall, K., Moran, T. P., and Halasz, F. G., "Tivoli: An electronic whiteboard for informal workgroup meetings," Proceedings of the InterCHI'93 Conference on Human Factors in Computer Systems. New York: ACM (1993).
While all of the foregoing systems are beneficial and useful in certain limited situations, they are all directed to solve limited problems. Accordingly, while they may be useful in an office setting, they might not easily transfer to other settings. Accordingly, there is a need for a document and information management system that is easier to use than traditional systems, yet powerful enough to be adaptable to numerous situations. Such a system should simplify the user's work, even if it does require some input and assistance. It should be able to handle documents and input in a variety of formats, structures, and media, including printed, written, and spoken communications.